Study Objectives

The goal of this paper is to explore the methodology of cross-validation and its application in evaluating the performance of predictive models for concrete strength. Concrete strength is a crucial parameter in construction, directly impacting the safety, durability, and cost-effectiveness of structures. Accurate prediction of concrete strength allows for optimal design, better resource allocation, and improved construction practices. Traditional methods of model validation, such as holdout validation, can sometimes provide misleading performance estimates due to their reliance on a single training-validation split. Cross-validation addresses this limitation by using multiple splits, thus providing a more robust evaluation of the model. As previously mentioned, cross-validation is a statistical technique used to assess the generalizability and reliability of a model by partitioning the data into multiple subsets, training the model on some subsets while validating it on others. This process helps prevent overfitting, ensuring that the model performs well on new, unseen data, and provides a more accurate estimate of the model’s performance.

Concrete strength is a crucial parameter in construction, directly impacting the safety, durability, and cost-effectiveness of structures. Accurate prediction of concrete strength allows for optimal design, better resource allocation, and improved construction practices. Traditional methods of model validation, such as holdout validation, can sometimes provide misleading performance estimates due to their reliance on a single training-validation split. Cross-validation addresses this limitation by using multiple splits, thus providing a more robust evaluation of the model.

In this study, we apply cross-validation to a dataset containing measurements of concrete strength. We aim to demonstrate how different cross-validation techniques, such as k-fold cross-validation and leave-one-out cross-validation, can be used to evaluate the performance of predictive models. By comparing these techniques, we seek to identify the most effective method for assessing model accuracy and reliability in the context of predicting concrete strength.

Model Measures of Error

Measuring the quality of fit of a regression model is an important step in data modeling. There are several commonly used metrics used to quantify how well a model explains the data. By measuring the quality of fit we can select the model that makes the most accurate predictions on unseen data. Common metrics used to measure model performance are:

• Mean Absolute Error (MAE)

The Mean Absolute Error is a measure error magnitude. The sine of the error does not matter because MAE uses the absolute value. Small MAE values, “lower magnitude” indicate better model fit. MAE is calculated (1) by averaging the absolute difference between the observed \((y_i)\) and predicted \(\hat{f}(x_i)\) values. Where:

• \(n\) is the number of observations,
• \(\hat{f}(x_i)\) is the prediction that the regression function \(\hat{f}\) gives for the ith observation,
• \(y_i\) is the observed value.

\[ \text{MAE} = \frac{1}{n} \sum_{i=1}^n |y_i - \hat{f}(x_i)| \tag{1} \]

• Root Mean Squared Error (RMSE)

The Root Mean Squared Error (2) is a measure of error magnitude also. Like MAE, smaller RMSE values indicate better model fit. In this method the square error \((y_i - \hat{f}(x_i))^2\) values are used. Squaring the error give more weight to the larger ones. In contrast with the MAE that uses the absolute error \(|y_i - \hat{f}(x_i)|\) values, all errors are weighted equally regardless of size. Taking the square root returns the error to the same units as the response variable, making it easier to interpret.

\[ \text{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{f}(x_i))^2} \tag{2} \]

• R-squared (\(R^2\))

R-squared is the percent of the variance in the response variable that is explained by the predictor variable(s). Unlike MAR and RMSE, \(R^2\) values range from 0 to 1 and the higher the value, the better the fit. An \(R^2\) value of 0.75 indicates that 75% of the variance in the response variable can be explained by the predictor variable(s). The \(R^2\) equation (3) is composed of two key parts, the Total Sum of Squares (\(SS_{tot}\)) and the Residual Sum of Squares (\(SS_{res}\)).

\[ \text{R}^2 = \frac{SS_{tot}-SS_{res}}{SS_{tot}} = 1 - \frac{SS_{res}}{SS_{tot}} = 1 - \frac{\sum_{i=1}^{n}(y_i - \hat{f}(x_i))^2}{\sum_{i=1}^{n}(y_i-\bar{f}(x_i))^2} \tag{3} \]

(James et al. 2013), (Hawkins, Basak, and Mills 2003), (Helsel and Hirsch 1993)

Leave One Out Cross-validations (LOOCV)

\[ CV_{(n)} = \frac{1}{n}\sum_{i=1}^{n} \text{Measuer of Errori}_i \tag{5} \]

Process: The steps for LOOCV are almost identical to k-fold cross validation. The only difference is that in K-fold, K must be less than the number of observations (n). In LOOCV, K = n, so when you split the data into testing and training data, the first testing fold is one of the observations and the training data would be every other observation Figure 2. In this way, every observation is tested against every other observation and the process would be repeated n times (James et al. 2013).

Data extraction, transformation and visulation

(I-C Yeh 1998) modeled compression strength of high performance concrete (HPC) at various ages and made with different ratios of components Table 1. The data used for their study was made publicly available and can be downloaded UCI Machine Learning Repository (I-Cheng Yeh 2007).

Load Libraries and Data

Data Plots

Data Correlation Plots

The Model

Construct the Linear Regression Model

Linear Regression: K-Fold Cross-validation

Linear Regression: Leave-one-out Cross-validation

Linear Regression Model: Nested Cross-validation

Construct LightGBM Model

LightGBM Model K-Fold Cross-validation

LightGBM Model: Leave-one-out Cross-validation

LightGBM Model Nested Cross-validation

Cross-validation Method Comparision

Conclution

In this study, we analyzed cross-validation techniques that can be used for evaluating concrete strength modeling performance, including K-Fold, leave-one-out cross-validation, and Nested cross-validation. In our case, we were able to examine the linear regression performance of an entire data set and then compare it with the performance of cross-validation techniques. The findings pointed out that leave-one-out cross validation, K-fold cross validation, and nested cross validation techniques had a better generalization error compared with conventional linear regression models. The detailed models established better results regarding the actual concrete strength. We can note that the nested cross-validation slightly performed better than the k-fold and leave-one-out cross-validation techniques. Further, the research also stressed the feature selection part, as variables like water-cement ratio, age, and type of aggregate were identified as main attributes influencing concrete strength. Overall, this kind of assessment raises the prospect of generating vehicle models for refining forecast precision and effectiveness, as well as investigating the best strategies for developing concrete mixes and promoting construction improvement.